{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = 'all'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 0,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 参数管理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 依靠深度学习框架管理操作参数：\n",
    "\n",
    "    * 访问参数，用于调试、诊断和可视化\n",
    "    * 参数初始化\n",
    "    * 在不同模型组件间共享参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "考虑一个单隐藏层的神经网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:39.200121Z",
     "iopub.status.busy": "2022-12-07T16:59:39.199802Z",
     "iopub.status.idle": "2022-12-07T16:59:40.341995Z",
     "shell.execute_reply": "2022-12-07T16:59:40.341206Z"
    },
    "origin_pos": 2,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.4204],\n",
       "        [0.4360]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 1))\n",
    "X = torch.rand(size=(2, 4))\n",
    "net(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 参数访问"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 5,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 当通过`Sequential`类定义模型时，可以通过索引来访问模型的任意层\n",
    "    - 这就像模型是一个列表一样，每层的参数都在其属性中"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:40.345895Z",
     "iopub.status.busy": "2022-12-07T16:59:40.345358Z",
     "iopub.status.idle": "2022-12-07T16:59:40.351780Z",
     "shell.execute_reply": "2022-12-07T16:59:40.350856Z"
    },
    "origin_pos": 7,
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "检查第二个全连接层的参数\n",
      "OrderedDict([('weight', tensor([[-0.1328,  0.0236,  0.2590,  0.2789,  0.3122,  0.0892, -0.2700, -0.0821]])), ('bias', tensor([0.2761]))])\n"
     ]
    }
   ],
   "source": [
    "print(f'检查第二个全连接层的参数\\n{net[2].state_dict()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 这个全连接层包含两个参数，分别是该层的权重和偏置\n",
    "    - 参数名称允许唯一标识每个参数，即使在包含数百个层的网络中也是如此"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- `state_dict()`是一个`Python`字典，将每层映射到对应的参数。一般下述对象拥有`state_dict`\n",
    "    - 包含可学习参数的层\n",
    "    - 优化器`torch.optim`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 9,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 访问目标参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 每个参数都表示为参数类的一个实例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:40.355017Z",
     "iopub.status.busy": "2022-12-07T16:59:40.354533Z",
     "iopub.status.idle": "2022-12-07T16:59:40.361115Z",
     "shell.execute_reply": "2022-12-07T16:59:40.360109Z"
    },
    "origin_pos": 11,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'torch.nn.parameter.Parameter'>\n",
      "================\n",
      "Parameter containing:\n",
      "tensor([0.2761], requires_grad=True)\n",
      "================\n",
      "tensor([0.2761])\n"
     ]
    }
   ],
   "source": [
    "# 下面的代码从第二个全连接层（即第三个神经网络层）提取偏置，提取后返回的是一个参数类实例，并进一步访问该参数的值。\n",
    "print(type(net[2].bias))\n",
    "print('================')\n",
    "print(net[2].bias)\n",
    "print('================')\n",
    "print(net[2].bias.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 14,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "- 参数是复合的对象，包含值、梯度和额外信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:40.364515Z",
     "iopub.status.busy": "2022-12-07T16:59:40.364074Z",
     "iopub.status.idle": "2022-12-07T16:59:40.370363Z",
     "shell.execute_reply": "2022-12-07T16:59:40.369347Z"
    },
    "origin_pos": 16,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 在上面这个网络中，由于还没有调用反向传播，所以参数的梯度处于初始状态\n",
    "net[2].weight.grad == None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 17,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 一次性访问所有参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 递归整个树提取每个子块的参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:40.373816Z",
     "iopub.status.busy": "2022-12-07T16:59:40.373095Z",
     "iopub.status.idle": "2022-12-07T16:59:40.379087Z",
     "shell.execute_reply": "2022-12-07T16:59:40.378031Z"
    },
    "origin_pos": 19,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('weight', torch.Size([8, 4])) ('bias', torch.Size([8]))\n",
      "=============================\n",
      "('0.weight', torch.Size([8, 4])) ('0.bias', torch.Size([8])) ('2.weight', torch.Size([1, 8])) ('2.bias', torch.Size([1]))\n"
     ]
    }
   ],
   "source": [
    "print(*[(name, param.shape) for name, param in net[0].named_parameters()])\n",
    "print('=============================')\n",
    "print(*[(name, param.shape) for name, param in net.named_parameters()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- `named_parameters()`返回一个遍历模型块的参数的迭代器，包含参数的名称与参数本身"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 21,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 这提供了另一种访问网络参数的方式\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:40.382676Z",
     "iopub.status.busy": "2022-12-07T16:59:40.381950Z",
     "iopub.status.idle": "2022-12-07T16:59:40.387675Z",
     "shell.execute_reply": "2022-12-07T16:59:40.386925Z"
    },
    "origin_pos": 23,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.2761])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.state_dict()['2.bias'].data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 26,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 从嵌套块收集参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 观察在多个块相互嵌套中参数如何命名？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:40.391302Z",
     "iopub.status.busy": "2022-12-07T16:59:40.390503Z",
     "iopub.status.idle": "2022-12-07T16:59:40.400904Z",
     "shell.execute_reply": "2022-12-07T16:59:40.400156Z"
    },
    "origin_pos": 28,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.3610],\n",
       "        [0.3610]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 定义一个生成块的函数（可以说是“块工厂”），然后将这些块组合到更大的块中\n",
    "\n",
    "def block1():\n",
    "    return nn.Sequential(nn.Linear(4, 8), nn.ReLU(),\n",
    "                         nn.Linear(8, 4), nn.ReLU())\n",
    "\n",
    "def block2():\n",
    "    net = nn.Sequential()\n",
    "    for i in range(4):\n",
    "        # 在这里嵌套\n",
    "        net.add_module(f'block {i}', block1())\n",
    "    return net\n",
    "\n",
    "rgnet = nn.Sequential(block2(), nn.Linear(4, 1))\n",
    "rgnet(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:40.404269Z",
     "iopub.status.busy": "2022-12-07T16:59:40.403645Z",
     "iopub.status.idle": "2022-12-07T16:59:40.407896Z",
     "shell.execute_reply": "2022-12-07T16:59:40.407140Z"
    },
    "origin_pos": 33,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequential(\n",
      "  (0): Sequential(\n",
      "    (block 0): Sequential(\n",
      "      (0): Linear(in_features=4, out_features=8, bias=True)\n",
      "      (1): ReLU()\n",
      "      (2): Linear(in_features=8, out_features=4, bias=True)\n",
      "      (3): ReLU()\n",
      "    )\n",
      "    (block 1): Sequential(\n",
      "      (0): Linear(in_features=4, out_features=8, bias=True)\n",
      "      (1): ReLU()\n",
      "      (2): Linear(in_features=8, out_features=4, bias=True)\n",
      "      (3): ReLU()\n",
      "    )\n",
      "    (block 2): Sequential(\n",
      "      (0): Linear(in_features=4, out_features=8, bias=True)\n",
      "      (1): ReLU()\n",
      "      (2): Linear(in_features=8, out_features=4, bias=True)\n",
      "      (3): ReLU()\n",
      "    )\n",
      "    (block 3): Sequential(\n",
      "      (0): Linear(in_features=4, out_features=8, bias=True)\n",
      "      (1): ReLU()\n",
      "      (2): Linear(in_features=8, out_features=4, bias=True)\n",
      "      (3): ReLU()\n",
      "    )\n",
      "  )\n",
      "  (1): Linear(in_features=4, out_features=1, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(rgnet) # 观察嵌套块结构和参数命名"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 35,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 因为层是分层嵌套的，所以可以像通过嵌套列表索引一样访问它们"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:40.410923Z",
     "iopub.status.busy": "2022-12-07T16:59:40.410613Z",
     "iopub.status.idle": "2022-12-07T16:59:40.416456Z",
     "shell.execute_reply": "2022-12-07T16:59:40.415718Z"
    },
    "origin_pos": 37,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.1391, -0.4768,  0.0063, -0.0645, -0.2130, -0.4114,  0.4821, -0.1601])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rgnet[0][1][0].bias.data # 访问第一个主要的块中、第二个子块的第一层的偏置项"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 40,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 参数初始化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 42,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "- 默认情况下，`PyTorch`会根据一个范围均匀地初始化权重和偏置矩阵\n",
    "- `PyTorch`的`nn.init`模块提供了多种预置初始化方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 45,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### 内置初始化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:40.419626Z",
     "iopub.status.busy": "2022-12-07T16:59:40.419352Z",
     "iopub.status.idle": "2022-12-07T16:59:40.426766Z",
     "shell.execute_reply": "2022-12-07T16:59:40.426002Z"
    },
    "origin_pos": 47,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Linear(in_features=4, out_features=8, bias=True)\n",
       "  (1): ReLU()\n",
       "  (2): Linear(in_features=8, out_features=1, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "权重为\n",
      "tensor([[-0.0098,  0.0099,  0.0122,  0.0087],\n",
      "        [ 0.0118,  0.0066, -0.0054, -0.0083],\n",
      "        [ 0.0024,  0.0002,  0.0026, -0.0210],\n",
      "        [ 0.0182, -0.0050, -0.0149,  0.0050],\n",
      "        [ 0.0084,  0.0126,  0.0110,  0.0083],\n",
      "        [ 0.0070, -0.0032, -0.0112,  0.0034],\n",
      "        [ 0.0050, -0.0016, -0.0045, -0.0128],\n",
      "        [ 0.0005,  0.0147,  0.0036,  0.0012]])\n",
      "偏置为\n",
      "tensor([0., 0., 0., 0., 0., 0., 0., 0.])\n"
     ]
    }
   ],
   "source": [
    "# 下面的代码将所有权重参数初始化为标准差为0.01的高斯随机变量，且将偏置参数设置为0\n",
    "\n",
    "def init_normal(m):\n",
    "    if type(m) == nn.Linear:\n",
    "        nn.init.normal_(m.weight, mean=0, std=0.01)\n",
    "        nn.init.zeros_(m.bias)\n",
    "net.apply(init_normal)\n",
    "print(f'权重为\\n{net[0].weight.data}\\n偏置为\\n{net[0].bias.data}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- ```python\n",
    "torch.nn.Module.apply(fn)\n",
    "```\n",
    "\n",
    "    - 将函数`fn`递归应用到**块的自身**和包含的**每个子块**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:40.430188Z",
     "iopub.status.busy": "2022-12-07T16:59:40.429543Z",
     "iopub.status.idle": "2022-12-07T16:59:40.436502Z",
     "shell.execute_reply": "2022-12-07T16:59:40.435761Z"
    },
    "origin_pos": 52,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Linear(in_features=4, out_features=8, bias=True)\n",
       "  (1): ReLU()\n",
       "  (2): Linear(in_features=8, out_features=1, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "权重为\n",
      "tensor([[1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1.]])\n",
      "偏置为\n",
      "tensor([0., 0., 0., 0., 0., 0., 0., 0.])\n"
     ]
    }
   ],
   "source": [
    "# 还可以将所有参数初始化为给定的常数，比如初始化为1\n",
    "\n",
    "def init_constant(m):\n",
    "    if type(m) == nn.Linear:\n",
    "        nn.init.constant_(m.weight, 1)\n",
    "        nn.init.zeros_(m.bias)\n",
    "net.apply(init_constant)\n",
    "print(f'权重为\\n{net[0].weight.data}\\n偏置为\\n{net[0].bias.data}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 55,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 对某些块应用不同的初始化方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:40.439840Z",
     "iopub.status.busy": "2022-12-07T16:59:40.439218Z",
     "iopub.status.idle": "2022-12-07T16:59:40.445909Z",
     "shell.execute_reply": "2022-12-07T16:59:40.445156Z"
    },
    "origin_pos": 57,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(in_features=4, out_features=8, bias=True)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Linear(in_features=8, out_features=1, bias=True)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "块0的权重为\n",
      "tensor([[-0.0491, -0.6622,  0.0968,  0.3570],\n",
      "        [ 0.5111,  0.4438,  0.4513, -0.6432],\n",
      "        [ 0.0939,  0.3542,  0.5616,  0.0479],\n",
      "        [-0.4754, -0.7070,  0.1619,  0.0199],\n",
      "        [-0.5642, -0.6589, -0.2447, -0.5487],\n",
      "        [ 0.5628, -0.3799, -0.2849,  0.4792],\n",
      "        [ 0.2969,  0.2397,  0.6815,  0.2212],\n",
      "        [-0.4634,  0.1852,  0.6426, -0.0074]])\n",
      "块2的权重为\n",
      "tensor([[42., 42., 42., 42., 42., 42., 42., 42.]])\n"
     ]
    }
   ],
   "source": [
    "# 下面使用Xavier初始化方法初始化第一个神经网络层，然后将第三个神经网络层初始化为常量值42\n",
    "\n",
    "def init_xavier(m):\n",
    "    if type(m) == nn.Linear:\n",
    "        nn.init.xavier_uniform_(m.weight)\n",
    "def init_42(m):\n",
    "    if type(m) == nn.Linear:\n",
    "        nn.init.constant_(m.weight, 42)\n",
    "\n",
    "net[0].apply(init_xavier)\n",
    "net[2].apply(init_42)\n",
    "\n",
    "print(f'块0的权重为\\n{net[0].weight.data}')\n",
    "print(f'块2的权重为\\n{net[2].weight.data}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 自定义初始化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 应对深度学习框架没有提供需要的初始化方法的情形"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 60,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "- 下面的例子使用以下的分布为任意权重参数$w$定义初始化方法：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "    w \\sim \\begin{cases}\n",
    "        U(5, 10) & \\text{ 可能性 } \\frac{1}{4} \\\\\n",
    "            0    & \\text{ 可能性 } \\frac{1}{2} \\\\\n",
    "        U(-10, -5) & \\text{ 可能性 } \\frac{1}{4}\n",
    "    \\end{cases}\n",
    "\\end{aligned}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:40.449458Z",
     "iopub.status.busy": "2022-12-07T16:59:40.448848Z",
     "iopub.status.idle": "2022-12-07T16:59:40.457344Z",
     "shell.execute_reply": "2022-12-07T16:59:40.456599Z"
    },
    "origin_pos": 66,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Init ('weight', torch.Size([8, 4])) ('bias', torch.Size([8]))\n",
      "Init ('weight', torch.Size([1, 8])) ('bias', torch.Size([1]))\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Linear(in_features=4, out_features=8, bias=True)\n",
       "  (1): ReLU()\n",
       "  (2): Linear(in_features=8, out_features=1, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=================\n",
      "初始化后的权重为\n",
      "Parameter containing:\n",
      "tensor([[-5.5998, -6.6353, -7.7560,  7.6029],\n",
      "        [-0.0000, -0.0000, -5.1531,  6.7816],\n",
      "        [-0.0000,  0.0000,  7.7268,  0.0000],\n",
      "        [ 7.2552, -0.0000,  0.0000,  8.7726],\n",
      "        [-0.0000,  0.0000,  6.4122,  0.0000],\n",
      "        [ 8.8125,  0.0000,  8.4305, -0.0000],\n",
      "        [-5.9873, -0.0000, -9.2604, -5.3338],\n",
      "        [ 7.6208, -0.0000, -0.0000,  0.0000]], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "# 自定义一个`my_init`函数来应用到`net`\n",
    "\n",
    "def my_init(m):\n",
    "    if type(m) == nn.Linear:\n",
    "        print(\"Init\", *[(name, param.shape) for name, param in m.named_parameters()])\n",
    "        nn.init.uniform_(m.weight, -10, 10)        # 用[-10,10]均匀分布初始化权重\n",
    "        m.weight.data *= m.weight.data.abs() >= 5  # 绝对值大于5保留，否则等于0\n",
    "\n",
    "net.apply(my_init)\n",
    "print('=================')\n",
    "print(f'初始化后的权重为\\n{net[0].weight}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 69,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 注意，始终可以直接设置参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:40.460372Z",
     "iopub.status.busy": "2022-12-07T16:59:40.460090Z",
     "iopub.status.idle": "2022-12-07T16:59:40.466428Z",
     "shell.execute_reply": "2022-12-07T16:59:40.465690Z"
    },
    "origin_pos": 71,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([42.0000, -5.6353, -6.7560,  8.6029])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net[0].weight.data[:] += 1\n",
    "net[0].weight.data[0, 0] = 42\n",
    "net[0].weight.data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 75,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 参数绑定"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 希望在多个层间共享参数：\n",
    "    - 可以定义一个稠密层，然后使用它的参数来设置另一个层的参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:59:40.469511Z",
     "iopub.status.busy": "2022-12-07T16:59:40.469232Z",
     "iopub.status.idle": "2022-12-07T16:59:40.477075Z",
     "shell.execute_reply": "2022-12-07T16:59:40.476334Z"
    },
    "origin_pos": 77,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.0084],\n",
       "        [ 0.0167]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 需要给共享层一个名称，以便可以引用它的参数\n",
    "\n",
    "shared = nn.Linear(8, 8)\n",
    "net = nn.Sequential(nn.Linear(4, 8), nn.ReLU(),\n",
    "                    shared, nn.ReLU(),\n",
    "                    shared, nn.ReLU(),\n",
    "                    nn.Linear(8, 1))\n",
    "net(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([True, True, True, True, True, True, True, True])\n"
     ]
    }
   ],
   "source": [
    "# 检查参数是否相同\n",
    "print(net[2].weight.data[0] == net[4].weight.data[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.0000e+02, -6.3952e-02, -1.0947e-01, -3.1406e-01, -2.9282e-01,\n",
       "         -3.4247e-01, -1.1970e-01, -3.2757e-01],\n",
       "        [-3.4288e-01,  3.3045e-02,  6.9978e-02, -1.3451e-01, -2.7620e-01,\n",
       "         -1.8093e-01,  3.1920e-01, -2.9720e-01],\n",
       "        [-2.5628e-01,  2.6267e-01, -6.5133e-02, -3.4902e-01,  5.3623e-02,\n",
       "         -1.4399e-01,  1.4990e-02, -2.6317e-01],\n",
       "        [ 1.8341e-01, -2.1717e-01, -2.3271e-01, -2.7987e-01,  2.5557e-01,\n",
       "          2.2681e-01,  1.2114e-01, -7.8814e-02],\n",
       "        [ 5.2703e-02, -2.7398e-01,  2.2674e-01,  2.4488e-01, -2.3777e-01,\n",
       "          2.9054e-01, -2.1716e-01,  1.8697e-01],\n",
       "        [-1.3274e-01,  9.1789e-02, -6.3458e-02, -1.8015e-01, -2.7990e-01,\n",
       "         -9.5318e-02,  1.9316e-01,  2.0636e-01],\n",
       "        [ 8.8571e-02, -7.5645e-02, -2.5454e-01, -2.1290e-01, -1.7103e-01,\n",
       "         -3.1130e-01, -3.9211e-02,  2.7355e-01],\n",
       "        [ 1.0877e-01, -3.5419e-03, -2.8627e-01, -3.4446e-01, -3.4175e-01,\n",
       "          2.1972e-01, -3.3592e-01,  8.8211e-02]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([True, True, True, True, True, True, True, True])\n"
     ]
    }
   ],
   "source": [
    "net[2].weight.data[0, 0] = 100\n",
    "net[2].weight.data\n",
    "# 确保它们实际上是同一个对象，而不只是有相同的值\n",
    "print(net[2].weight.data[0] == net[4].weight.data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 81,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "- 上面例子表明第三个和第五个神经网络层的参数是绑定的，它们不仅值相等，而且由相同的张量表示。\n",
    "- 因此，如果我们改变其中一个参数，另一个参数也会改变"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "幻灯片",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
